Comparison of ontology alignment algorithms across single matching task via the McNemar test

Ontology alignment is widely used to €nd the correspondences between di‚erent ontologies in diverse €elds. A‰er discovering the alignment by methods, several performance scores are available to evaluate them. Œe scores require the produced alignment by amethod and the reference alignment containing the underlying actual correspondences of the given ontologies. Œe current trend in alignment evaluation is to put forward a new score and to compare various alignments by juxtaposing their performance scores. However, it is substantially provocative to select one performance score among others for comparison. On top of that, claiming if one method has a beŠer performance than one another can not be substantiated by solely comparing the scores. In this paper, we propose the statistical procedures which enable us to theoretically favor one method over one another. Œe McNemar test is considered as a reliable and suitable means for comparing two ontology alignment methods over one matching task. Œe test applies to a 2 × 2 contingency table which can be constructed in two di‚erent ways based on the alignments, each of which has their own merits/pitfalls. Œe ways of the contingency table construction and various apposite statistics from the McNemar test are elaborated in minute detail. In the case of having more than two alignment methods for comparison, the family-wise error rate is expected to happen. Œus, the ways of preventing such an error are also discussed. A directed graph visualizes the outcome of the McNemar test in the presence of multiple alignment methods. From this graph, it is readily understood if one method is beŠer than one another or if their di‚erences are imperceptible. Our investigation on the methods participated in the anatomy track of OAEI 2016 demonstrates that AML and CroMatcher are the top two methods and DKP-AOM and Alin are the boŠom two ones. Moreover, the Levenstein and N-gram string-based distances discover the most correspondences while SMOA and Hamming distance are the ones with the least found correspondences.